Model transform docs #1665
Model transform docs #1665
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1665
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 8676b86 with merge base 30b8519 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
| Message( | ||
| role="user", | ||
| content=[ | ||
| {"type": "image", "content": Image.new(mode="RGB", size=(224, 224))}, |
There was a problem hiding this comment.
I love this entire file! very good job!
any chance that the example here can have more than 1 image?
| def __call__( | ||
| self, sample: Mapping[str, Any], inference: bool = False | ||
| ) -> Mapping[str, Any]: | ||
| # Expected input parameters for vision encoder | ||
| encoder_input = {"images": [], "aspect_ratio": []} | ||
| messages = sample["messages"] | ||
|
|
||
| # Transform all images in sample | ||
| for message in messages: | ||
| for image in message.get_media(): | ||
| out = self.transform_image({"image": image}, inference=inference) | ||
| encoder_input["images"].append(out["image"]) | ||
| encoder_input["aspect_ratio"].append(out["aspect_ratio"]) | ||
| sample["encoder_input"] = encoder_input | ||
|
|
||
| # Transform all text | ||
| sample = self.tokenizer(sample, inference=inference) | ||
|
|
||
| return sample |
There was a problem hiding this comment.
shouldnt there be some reference here to the decoder input?
There was a problem hiding this comment.
Maybe I should add some lines showing how a custom transform would be used
| self.tokenizer = llama3_tokenizer(...) | ||
| # Image transforms | ||
| self.transform_image = CLIPImageTransform(...) | ||
| self.xattn_mask = VisionCrossAttentionMask(...) |
There was a problem hiding this comment.
nit: fine to ignore, but should we import those for example completion?
There was a problem hiding this comment.
mm this is just to show a snippet of what a transform may look like, not intended to be runnable
| @@ -27,7 +27,7 @@ class FlamingoTransform(ModelTokenizer, Transform): | |||
|
|
|||
| Args: | |||
| path (str): Path to pretrained tiktoken tokenizer file. | |||
| tile_size (int): Size of the tiles to divide the image into. Default 224. | |||
| tile_size (int): Size of the tiles to divide the image into. | |||
There was a problem hiding this comment.
is that true? did we remove the default?
There was a problem hiding this comment.
yep, this is a required argument
| role="user", | ||
| content=[ | ||
| {"type": "image", "content": Image.new(mode="RGB", size=(224, 224))}, | ||
| {"type": "text", "content": "What is in this image?"}, |
There was a problem hiding this comment.
nit: probably worth a link somewhere here on how to build a dataset. Fine to ignore, since its probably to find it in the docs anyway
Context
What is the purpose of this PR? Is it to
Please link to any issues this PR addresses.
Changelog
What are the changes made in this PR?
*
Test plan
Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.
pre-commit install)pytest testspytest tests -m integration_testUX
If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example